Even these ideas are not so novel. For example, the local reparametrization trick is something that we use all the time when we do Variational Bayes (VB) (say in a logistic regression model) and transform high-dimensional integrals into one-dimensional integrals under a Gaussian approximate posterior. For example, if you have a likelihood of the form \prod_{i 1} n \sigma(w T x_i) and apply VB with q(w mu,Sigma), then you end up with a sum of expectations of the form \sum_{i 1} n q(w mu,Sigma) \log \sigma(w T x_i) d w and then the local reparametrization trick is applied to transform each separate (initially high-dimensional integral over the vector w) into a 1-D integral over the univariate standard normal. The authors essentially use this separately for each activation unit and apply stochastic approximation instead of integration. Having said that, I must admit that as far as the stochastic variational inference algorithms are concerned and the related research community (born a couple of years ago!) the use of this local reparametrization trick, as far as I know, is novel and people should know about it because it is useful.